As of February 9, 2024, Amazon Kinesis Data Firehose has been rebranded to Amazon Data Firehose. For further details, you can read the AWS What’s New post. This platform now offers a simplified and user-friendly method to stream data to external services for analysis, including log data from various AWS services. Given the diverse formats and substantial volume of this data, it presents a formidable challenge to recognize and connect critical event details to resolve issues and enhance application performance. The new capability allows for the seamless transmission of logs from AWS services directly to Datadog for analysis. Since Kinesis Data Firehose is fully managed by AWS, there is no need to handle additional infrastructure or forwarding configurations for log streaming.
In this article, we will explore the integration of Kinesis Data Firehose with Datadog and illustrate how to ingest logs from multiple AWS services into Datadog. Kinesis Data Firehose facilitates real-time streaming data delivery to destinations such as Amazon S3, Amazon Redshift, or Amazon OpenSearch Service, and now includes support for sending data streams to Datadog. It provides built-in, fully managed functionalities for error handling, transformation, conversion, aggregation, and compression, eliminating the need for custom applications to manage these complexities. These features empower users to monitor and analyze data from any source, thereby delivering operational intelligence and enhancing business performance.
Datadog, recognized as an AWS Partner Network (APN) Advanced Technology Partner with AWS Competencies in DevOps, Migration, Containers, and Microsoft Workloads, allows users to delve deeper into log analysis and gain valuable insights into the status of their applications and AWS infrastructure. You can evaluate all your AWS service logs while retaining only the necessary ones, generating metrics from aggregated logs to identify and alert on trends in your AWS services.
How It Works
To process and deliver AWS service logs to Datadog, you can utilize the following architecture.
By implementing this solution, you can configure AWS Web Application Firewall (AWS WAF), Amazon Route 53 Resolver Query Logs, or Amazon API Gateway to send log events directly to Kinesis Data Firehose. If you are using a different service that outputs logs to Amazon CloudWatch Logs, you can use CloudWatch log subscriptions to channel log events from CloudWatch Logs into a Firehose delivery stream. By setting up Kinesis Data Firehose with the Datadog API as a target, you can forward the logs to Datadog for comprehensive analysis. Additionally, you can back up events to an S3 bucket to mitigate against data loss. You have the option to back up all log events or only those that exceed a specified retry duration. For more information on adjusting your backup according to the volume of log data being streamed, refer to the Amazon Kinesis Data Firehose Quota.
Creating Your Delivery Stream
To successfully send logs to Datadog, your new delivery stream will require an API key, which can be located in the API section of your Datadog account.
If you do not possess an account, you can visit the Datadog website to register for a free 14-day trial. To establish your delivery stream, follow these steps:
- On the Kinesis Data Firehose console, select Create delivery stream.
- For Delivery stream name, input a name.
- For Source, select Direct PUT of other sources.
- Click Next.
- On the Process records page, keep all settings at their default and click Next.
- On the Choose a destination page, select Third-party partner for Destination.
- From the Third-party partner drop-down list, select Datadog.
- For the HTTP endpoint URL, choose the appropriate logs HTTP endpoint based on your Region and Datadog account configuration. For more detailed guidance, see their Logs Guides.
- Enter your Datadog API key in the API key field to enable your delivery stream to publish to the endpoint.
- Select GZIP for Content encoding.
- Retain the default settings for Retry duration.
- For S3 backup mode, choose Failed data only.
- Enter the S3 bucket for delivering log events that exceeded the retry duration. You can also create a new bucket by selecting Create new.
- Click Next. Accept the default Datadog and Amazon S3 buffer conditions for your stream in the Datadog buffer conditions section.
- In the IAM role section, configure permissions for your delivery stream by selecting Create or update IAM role.
- Click Next. Review your settings and select Create delivery stream.
Logs that are subscribed to your delivery stream will now be available for analysis using Datadog.
Conclusion
Datadog enables the monitoring of servers, containers, databases, and third-party services to ensure high availability, optimize performance, and troubleshoot issues. With Kinesis Data Firehose HTTP endpoint delivery, AWS service log data is readily accessible for analysis, allowing for the identification of issues and performance bottlenecks in applications by correlating logs from AWS services such as AWS CloudTrail, Amazon RDS, and AWS Lambda functions with metrics and traces. Utilizing a fully managed AWS service for this delivery method guarantees high availability and scalability for your integration. For additional information on configuring this integration directly via AWS CloudFormation, visit Log management. To learn more about using HTTP endpoint delivery with CloudWatch metrics, see the excellent resource on onboarding new hires during COVID-19.
About the Authors
Alex Thompson serves as the Vice President of Product and Community at Datadog.
Jacob Lee is a Senior Technical Account Manager with AWS, dedicated to assisting clients in migrating their workloads to the cloud. In his leisure time, he strives to keep up with his 2-year-old.
Emily Carson is a Senior Technical Account Manager with AWS, focused on helping large enterprises construct secure and scalable solutions on the AWS cloud. In her free time, she enjoys reading classic science fiction and creating quirky projects with her family.
Leave a Reply